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ABSTRACT 


An examination of the Wald Stopping bounds ore. 
Sequential Probability Ratio Test (SPRT) is made by compar- 
ing results obtained from Monte Carlo simular ionomers 
sequential sampling tests with results obtained using 
Wald formulations. Operating Characteristic, ASN} ean 
V{N] values are presented for tests sampling from each of 
eight Binomial, 14 Exponential, and 24 Normal distribu- 
tions. An extensive bibliography of references associa. a 
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Pee BU ECTIVE 


In this thesis the accuracy of the Wald approximate 
Peers On limits Of the Sequential Probability Ratio Test 
(SPRT) are investigated. The Wald approximate limits are 
compared with "exact" emperical limits obtained by Monte 
ear lo simulation. An extensive bibliography of references 


associated with the SPRT is included. 


Il. HESTORY OF OERT 


In statistical theory the size of a sample may oOreme 
not be fixed prior to observation of certain sample Valijea 
If, in a test of hypotheses, the sample size 2s MoU eee 
in advance, the decision to terminate sampling may depend 
upon the values of the previous samples. Such a test is 
said to be sequential. 

The first mention of sequential test procedures was 
by H. F. Dodge and H. G. Romig who, in 1929, construcved 
a Double Sampling Plan. Prior to World War II, thereware 
not many entries in the literature concerning sequential 
procedures. During World War II the Statistical Research 
Group of Columbia University operated under a contract 
with the Office of Scientific Research and Developement 
and was directed by the Applied Mathematics Panel of the 
National paver ee Research Committee. Milton Friedman 
W. Allen Wallis. members of the research group, recognized 
the great potentialities and far reaching consequences 
that sequential analysis might have; consequently various 
members of the group, and in particular A. Wald, worked out 
what is known as the SPRT. In the early 1940's manyerue 
these results were classified, however, the Restricted 
classification was removed in 1945. 

Abraham Wald, in 1943, worked out the basic principles 
for the SPRT [Wald, 1943]. During the next two years wyaua 


continued working on the basic principles @f tie sora 


1G 


Pie liaiiewametencra, Consideration of cumulative sums of 
independent random variables which gives the Operating 
Characteristic (OC) curve of any SPRT, and the character- 
Parent unech1on Of The number of observations required by 
the test [Wald, 1944]. 

Duirine Choiswseme period of time, independent work on 
sequential inferences was also conducted in England by 
G. A. Barnard [Barnard, 1946] who derived general results 
Similar to those obtained by the Statistical Research 
Group at Columbia. 

Since the unfortunate death of A. Wald in 1950, there 
have been many varied contributions Loe ciowliteralrure of 
sequential methods. These contributions tend to deal 
beamerily wich specific families of distributions. Many 


of the articles are listed in the bibliography. 


1. 


Til. THEORETICAL. BACKGROUND OF Ah Sere 


The Sequential Probability Ratio Test of a simple Null 
Hypothesis against a simple Alternate Hypothesis differs 
from fixed sample size hypothesis tests in that it is 
conducted in stages, where a stage constitutes evaluation 
of an observation. At each stage one of three alternatives 
is chosen: (1) discontinue sampling, accept the null 
hypothesis; (2) discontinue sampling, reject the null 
hypothesis; (3) draw another observation. The procedure 
continues until one of the alternatives (1) or (2) is 
chosen. Under quite general assumptions, the probability 


of eventual termination of the SPRT is equal to one.t 


A. DEFINITION OF SPRo 
Let the distribution of the random variable, 7 etme os 
consideration be given by the density or mass function, 


DUx 2 6) ee et H, be the Null Hypothesis that 6 = oF and 


Hy be the Alternate Hypothesis that 6 = 8, > 80° Therefore 


the density or mass function of (ow pe mn ed. 8.) when 


H, is true and Pex: 6,) when H, is true. Successive 


il 


independent observations on X will be denoted Xs 


doe Ree saseeee . The SPRT is based on the likelihood avis 
iar n P(x, 3 8,) 
= i=1 7 ae 9 
1 


A. Wald, Sequential Analysis, Wiley, N.Y., 1947, p.157. 


We 


and two positive numbers A and B, A>l and B<l. After 
each observation on X, the procedure for choosing one of 


the three alternatives is: 


iit <8; Discontinue Sampling, Accept He 
2 ak 1 2A, Discontinue Sampling, Accept Hy 


Boat B<IL_<A; Draw another observation. 


be DERIVATION OF STOPPING BOUNDS 

The two constants A and B are determined so that the 
test will have (nearly) the prescribed probabilities, 
a and g, of making errors, where a is the probability of 
making a Type I error (Rejecting A, when it is true) and 
8 is the probability of making a Type II error (Accepting 
au when it is false). Exact values for A and B could, in 
principle, be obtained from the following equations given 


is 


mene first that He is true and in the second that Hy 


true: 


Q 
if] 


P[l, 2A] GE P[I,>A, B<Il, <A] eure netate si 


cee) 
Il 


ieee Sills, Bal <A} + ...... ae: 


iMmeGarce rece, approximations for A and B developed by 
Wald are usually used, where a and 8 are specified apoiri.* 


The Wald approximate stopping bounds are 


il 


> 
l 
| 
D 





99) 
Il 
WL 





26 Wald, Sequential Analysis, op.cit., pp. 40-44. 
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The boundaries used in the tests performed for this 
thesis were formulated as two diagonal lines with proper 
intercepts. These boundaries are called the acceptance 
number, Ny and the rejection number, Rae These numbers 
are obtained by setting the logarithm of I> the likeli- 
hood ratio, equal to the logarithms of A and B. The 
resulting test is the Wald SPRT with stopping bounds A 


and B. 


C. RANDOM WALK OF THE OBSERVATION RESULTS 

At each observation the value of the test statisvume 
was tabulated. The stepwise values of the test statistics 
can be graphed with the abscissa being the number, nygas 
observations made and the ordinate being the value of the 


test statistic. The boundaries, A, and Rep will limitwe@e 


n 
steps of the random walk. The test terminates when either 
boundary is reached or surpassed by a value of the test 


Stacristic. 


D. THE OC FUNCTION Cl Titec no 

The Operating Characteristic (OC) Function, L(6), is 
defined as the probability that the sequential test will 
lead te accepiancerson ie when @ is the true value of the 


parameter. Using the approximations on the stopping 


3a. Wald, "Sequential Tests of Statistical Hypothesis," 
Annals of Math. Stat., Vol. 16, No. 2, 1945, pp. egos 
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bounds mentioned above, Wald mrcied:: tiagarnesOG Tunct lon 


could be approximated by 


n(@) 
- oA = 
L(@) = Gy 
ul) GD 


where h(6@) is a non-negative real number such that 


eo fe: 65) ig 39) 
i eC 6)" meqmuNedx = de, (2) 


=_ CO 


or the equivalent summation in the case f is a mass function. 
It is sometimes difficult in practice to obtain the 

value for h(8) for each of various given values of 6, in 

such cases a "reverse" process may be used: set h(6) 

equal to a non-zero real number and compute a corresponding 

value of 6. This technique was used in computing L(6) for 


the Binomial and Exponential distributions in this thesis. 


Pelee Cle SAMPLE SIZE OF THE SPRT 

As mentioned above, with probability one the SPRT 
eventually terminates. Thus, using the approximate boun- 
daries, A and B, and disregarding the "excess" of IL over 


these boundaries at termination, 


Pin. B 


Z |e) = L(é@) 


P{Ln A = Z| 6] = 1 - L(6) 


ve Wald, Sequential Analysis, op.cit., pp. 48-52, 
161-64. 


1? 


N N 
where Z = 2% Ln I =r Zs and where in turn N is the 
i=l i=1 


Sample size required for termination. Therefore, the 


conditional expected value of Z, given 6, 1S approximatvely 


E(Z| 6) = (1-L(6)) LnA + L(@)Ln B 


> 


Utilizing Wald's Fundamental Identity,° the condi uaemam 


expected value of Z, given 8, can be written 


E(Z | 6) = Ef 


tums 
= 


Z, | @] =[E]N E(Z, | 6) 
i 
The Expected Sample Size for the test is therefore given 


approximately by 


- E(Z| 6]. © (lene nee eee 
E[(N] = tT oT > fC ae . (3) 


Tee E[Z, | 6] = 0, one may approximate the Expected Sample 


DiZe TDy 


EIN] = Eze een 


2 
E[Z, een 
F. VARIANCE OF SAMPLE SIZE OF THE SPRY. 


It would appear that the Variance of the Sample Sie 


(N), could be approximated in certain cases using an 


A, Wald, Sequential Analysis, op.cit., pp. 159mm 


a0 Wald, "Differentiation under the Expectation Signa 
the Fundamental Identity of Sequential Analysis," Annals of 
Math. Stat., Vol. 17, No. 4, p. 47235 becemeer ee 
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@pproach Similar to that for E[N]. During his literature 
Peview the author encountered only three references to 
such an approximation [Wald, 1945a], [Walker, 1950], 

[Cox and Roseberry, 1966a]. 


Since 
VIN] = E[N°] - E°[N] (4) 


an approximation for V[N] may be developed as follows: 
B“(N] can be approximated using Eq. (3). An approximate 


value for the E[N°] may be derived from 


9 N 
mee yee) = ELC & Z.) 


UtiliZing Wald's Fundamental Identity, this may be written 
2 _ 2 2 
E([Z{ 6) = E[NJE(Z,°| @] + E[N(N-1)]E°[2Z,| 6] 


Vaeewe., as before, Zs and Zs are independent and identically 


distributed. Expanding and collecting terms 


E(Z“| @] = E(NI(E(Z,°| 0] -E°(Z,| 01) + E(N°1E°IZ, [6] 
E(Z“| 6] = E[N] vtz,| 6] + E{N°] E°(z,| 61 
Then 
2 
Eee ieee INI VizZ.| 6) 
eC (5) 
E (Z, | 6] 


ale? 


with 
2 ; 
E{z’| 6] = [1-L(@viEe° s ea@eninee 
Using Eqs. (3) and (5) in (4), an approximate Variane= 
of the Sample Size is thus given by 


E[Z°| 8] - E(N]V{Z,| 6] - E°{Z| 6] 


V[N] 
ogi 6] 


V{z| 6] - E[N] VIZ, | 6] 
5 Cee 


Mx E°(z,| 0] 
al 
Unfortunately, using Eq. (6) to calculate the approxi- 

mate Variance of Sample Size leads to negative values in 

many cases. This appears to be caused by the magnification 
of the approximations used in Eq. (3) when they are entered 
in Eq. (5). Whether this is in fact the true cause, the 
approximation in general is met goed. Consequently) me 
numerical tabulations of the V[N]Jusing Eq. (6) are included. 
However, emperically determined "exact" values of the V[N]} 


are included. 


as 


ero eoteroUL TONS INVESTIGATED 


ieseoulbnorminvestieatred the SPRI for three common 
Petry Oupions: the Binomial, Normal, and Exponential. 
Each distribution was investigated for various parameter 
Poues. fachespec fic distribution was used in calcu- 
lating the Wald "approximate" results and in generating 
empirical "exact" results for the OC curve, Expected 
Pape oize, and the Variance of Sample Size. 

Emplicit equations for the points on the Wald “approx- 
imate" Operating Characteristic Curve, the "approximate" 
Expected Sample Size Curve, and the "approximate" Variance 
mee cplcm. ze Curve are given for these distributions using 


jew (1), (3), and (6). 


Poe DINOMTAL DISTRIBUTION 


1-x 


Suppose f(x; 0) = @* (1-6) the logarithm of the 


likelihood ratio is 


9 Es oe ae a 
1 1 
ae te Galea |: 
O Oo 
Then 
EGle 8, ) iGO 8,) 


E(Z, | 6) = 6Ln ae, 85) + (1-6) in CONT) 





1-9 
5 4 Gee tal sg? | (7) 
O 


Ihe 


and 


VIzZ,| 6] = E[Z,“| 8] + B°(Z,| 2) 
= Ne 
24] 6) = e(1ee) [un —2 (8) 
VipZas lee ole ~ ca : 
i . coe 


In order to obtain an approximate OC function, the 


expression of Eq. (2) for this case, 


Be (eet 
al eat 

7 Gis) a =1, 
Aa Ci om 


was solved for 86 and evaluated with various selected values 


of nCey- 
i hice> 
Le (1-6, ) 
ees) 
9 : : (9) 
aan Cle: r Tale} 
a (1-8), 
x, | ~ 1-0) 
_ O i J 


Utilizing Eq. (9) with Eq. (1) points on the Wald a@pgmeaaee 
mate OC Curve were obtained. 

By substituting Eq. (7) into Eq. (3) the Wald Wapgmgerames 
mate Sample Size curve was determined. An attempt was 
made to determine the approximate Variance of Samplewsuzee 
utilizing Eqs. (7) and (8) with the prior results Of 
in Eq. Col 


The acceptance and rejection numbers were respectively 


computed from 


20 


Mea) 


O 
fers se min 
i (1-6,) 
ig s) (TSR 
Ln i - Ln 1 
S) (1-6) 
O O 
and 
(ilo 
tio A + nin 
(izes 
os 2. aa 
a ae 
ae ((1-8,) 


B. NORMAL DISTRIBUTION 


itemueamc eis Normal (6, se) then the log likelihood 





ratio is 
ee! ee 
vo a= [2(6,-8,)X,+ Gc -8) Dk 
and 
Q e_¢ e(e@.-e@ ) 
ie) All JES) 
E(Z, | 6] = Se i a (10) 
20 fe} 
and 
(8,-8,)° 
V(zZ.| 6) =§$ — (ae) 
ail of 


pbslbivucime Bq. (10) into Eq. (3) points on the Wald 
approximate Sample Size Curve were computed. Eqs. (10) and 
(11) with E[N] computed above were used in Eq. (6) in an 
attempt to determine the approximate Variance of N. 

In order to determine points on the approximate OC 
curve, the expression of Eq. (2) for the present case was 
Solwea for (6), resulting in 


eal 


h(@) = 


The acceptance and rejection numbers are, in this eae 


respectively given by 








a a 
AL = [o /(8,-8.)]Ln Bla Jat | 5 ] 
and 
@ +80 
a 2 7 Onue 
R, = [o /( 8, 6.) Jin Ace ot 5 } é 


C. EXPONENTIAL DISTRIBUTION 


If f(x; 6) = 6 e °*; x>0, the log likelihood ratio 


is 
eT 
BS Na = (0,-8,)%, 
O 
so that 
6 Co 20m} 
E A oe 
E[Z, | Oss Un ee 5 (iz 
O 
and 
(8,-6_)¢ 
V2 ae. (13) 
6 


Substituting Eq. (12) into EqQl ys) coimus con therieum 
approximate Sample Size curve were obtained. Eqs. (12) and (13) 
were used in Eq. (6) in an attempt to determine an approxi- 


mate Variance of N. 


ee 


In order to determine values for an approximate OC 


curve, Eq. (2) was solved for 6, resulting in 


neo) (8,-8.) 
§ = =a ° (14) 


fel = 
6) 


Merlizing Bas. (1) and (14) points on the Wald approximate 


OC Curve were obtained. 


The acceptance and rejection numbers are respectively 


given by 
oF) 
-in B + nbn a 
is = Eee) 
n 
(6,-98.) 
and 84 
Ln A + nbn 5. 
— = 
n 


(8,-8.) 


a3 


V. PROCEDURE 


Computer programs, coded in FORTRAN IV, were written 
which gives points on the Wald approximate OC Curves, and 
the approximate Expected Sample Size curves. Programs 
for the Binomial, Normal, and Exponential distribuvienae 
each entitled "Approximate (Distribution)" are listed under 
Computer Programs. 

In order to evaluate the Wald approximations, companion 
programs called "exact" programs were written. These 
programs produce Monte Carlo simulation of the SPRT pro- 
cedure for the distribution considered. Each program 
determines "exact" points on the OC curve, Expected Sample 
Size curve and Variance of Sample Size curve. 

The simulation models sequential sampling from a 
specific known distribution with parameter values being 
inputs to the simulation. As each sample was observed, a 
corresponding acceptance and rejection number was computed 
and the test statistic was computed. These values were 


compared and the appropriate alternative was selected: 


A. DETERMINATION OF THE NUMBER OF REPLICATIONS 

In order for the simulation estimates of the Operaviig 
Characteristic pointe. L(@), (a Bernoulli parameter) to be 
useful, it is necessary to use a large number of replica— 
tions (that is, simulate the performance of many tests) so 


that with a high probability L(8) is “elose” to tiemtmne 


ay 


L(@). A Normal approximation to the binomial was used to 
determine the number of replications that should be used 


at each sample point. This was done as follows: 


mae ewachove the number of successes in J independent 


J 
repeated Bernoulli trials with parameter p, and let 


denote the average number of successes in J trials. For 
large J, Ps can be shown to be approximately Normal with 
mean p and variance p(l-p)/J. Then Y= (P;-p)/Ip(1-p)/J1* 
is approximately Normal (0, 1). For any level of risk, 
y>0O, and minimum acceptable probability bound, 6>0, we 


seek J such that 





P 
J Y 
eee eee | S| > 
Yp(I-p)/d Yp(1-p)/J 
or 
) wiH<y,< ee fe), 
peal!) /pCi=p)| 
This occurs whenever 
2 
as NES ptiep) 
y 
: _ 1-6 ; : 
where ys is such that P(Y<-y .] ane’ where Y is dis- 
2 
Pew vecmicrnal (0, 1). At p= 1-p #= 1/2, J > ie SO 
ey, 
for y = .01 and 6 = .95, Y, = 1.645 and the required 


6 


eae 


valu@ af J i8 approximately 6, /O5 ee deteriora 


po ee Yo= Od 6 = .95 requires J = 1481; 
Pos Oe eee 6 = .95 requires J = 163; 
Bo ale y= 2 Ce OS 905 requires J = 2095; 
0. =a oe vy = eee 6 = .95 requires J = 4889. 


In view of these and similar determinations and sincewam 
was desired to estimate L(6@) values as large as .30 to 
within reasonable accuracy, a selection of 5000 replica= 
tions for each estimation of L(8@) value was made. The 
values obtained in the simulation should therefore, with 
high probability, be accurate to at least 2, and usualy 
3, decimal places. A fourth decimal was carried iiieeee 


tables to exclude oeseis Ve Wo Uieen ase) erce, 


B. COMPUTER SiIMUDATiICy 
For each distribution investigated two computer progaas 
were developed; an "approximate program" based on the Wald 
approximations for the SPRI, and the “exact  vrogramn) 
involving a Monte Carlo simulation. The inputs necessary 
for these programs are the number of distributions to be 
inspected, the parameter values for the Null and Alternate 
iypetheses (8. and 64); the “target" Type I and Typewim@ 
“ror Probabilities (oa and 8), parameters for the distiaaam. 
boing sampled (@), arguments for the random number 
nerator (URN, a special random number generator included 
hee [BM Seientific Package at the Naval Postgraduate 


, the number of replications (in this thesis 5000) 


26 
c 


Btryeachn test point, (6), and the number of test points 
from the "approximate program" being used in the "exact 
meeeram.  Lhese values, punched in one input card, and 
the associated output from the "approximate program," 
Haeen 1s run first, comprise the input data for the exact 
program. 

simulation of observations from Binomial, Normal, and 
Exponential distributions were by standard methods. Back- 
ground information may be obtained in [Naylor, 1967] and 
[McMillan and Gonzalez, 1968). 

in order to obtain the values tabulated in this thesis 
the procedure was to first run an "approximate program." 
ieeeee programs produce a fixed number of test points. In 
the Approximate Binomial program the number of test points 
(now 85) is controlled by the last "IF" statement in the 
program, varying the value 41 changes the number of test 
Points. Control of the number of test points (80) in the 
Approximate Normal program is by changing the numerical 
Wberoliers an the first "XM2" and "XM" steps. Control 
of the number of test points (50) in the Approximate Expo- 
ential program is by changing the limit on the "DO 7" 
Statement. The presentation of the Approximate Program 
output (Approximate OC Value, 6, Approximate Expected 
Sample Size) is a listing and a separate punched card for 
SJeamvesvepolie, ©. Prior to running the "exact program," 


the number of test points was reduced to 15 to 20 so as to 


eit 


reduce the execution time. The Exact Program output (see 
tables) is a listing and a set of punched cards: 

"Exact" OC values of various SPRT's were estimated by 
the relative frequency of the number of replications ter- 
minating with acceptance of Ho» at each of several values 
Ciel 6 

"Exact" expected sample sizes at each test point, 6, 
were estimated by tallying, each time the test was performed, 
the observed sample requirement. The average sample size 
being computed at the end of the 5000 replications. These 
averages were taken to be the exact values of E[N]. 

The "exact" variance of sample size at each test point, 
8, was computed in a manner similar to that used for The 


Bale 


m,° - nE* {N] limes 


ae se) 


VIN] = 7 I 


i=l 


where n = 5000 replications and ms denotes the number of 


observations required in the ith simulated Cesc. 


C. COMPUTER STATISTICS. 

The IBM-360 at the Naval Postgraduate School was 
utilized in computing the values presented. The maximum 
core space needed for any one of the programs was less 
than 58K bytes. The necessary time to complete an 
"approximate" case was less than 5 seconds. The "exact" 


programs were run under the H compiler of the IBM-360. In 
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S eeatiecases tits allowed an execution time savings of 
80% with respect to running under the G compiler. The 
Pmownieesorme per case for the programs averaged 11 minutes 


as presented, with a maximum of about 15 minutes. 


2s) 


VI. CONCLUSIONS AND RESULTS 


A comparison of the Wald "approximate" results with 
the "exact" results described above confirmed the known 
fact that, in all cases, the results obtained using the 
Wald approximations for the SPRT are conservative in that 
a given test plan's error probabilities are greater than 
the exact values. In the Binomial cases as much as a 
O2% difference in the target a level and the exact a 
level was noted; for the Exponential distribution in one 
case Oa, =n 0:. H, = 5.0, 0 = .03, 8 = 2.05) ay5o7 
difference was noted, with the differences generals 
averaging about 25 to 30%. For the Normal distribution 
the percentage difference in the target a level and the 
exact a level appears to increase with |, - 6, |. 

A comparison of the approximate and exact expected 
Sample sizes is facilitated by the tabilized values. In 
all cases the results following indicate that at eachmmees 
point the "exact" expected sample size is larger than the 
Wald approximation. 

The exact variances on Sample size found here support 
the conjecture of [Cox and Roseberry, 1966a], namely, 
that V[N] is approximately the square of E{N]). Unfor- 
tunately, as noted earlier a natural approximation for 
V{(N] omnes out to yield especially poor approximations. 
However, the exact variance of N does appear to increase 


roughly as the square of E[N], in the cases considered here. 
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imeem rOorvy=six Cases were examined. The final 


results of all cases are tabulated below. 
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